Testing is a measure to provide insight into the quality of delivered products and the related risks when taken into production. If the quality is inadequate timely measures can be taken, such as rework by the developers. However, there is never an unlimited quantity of test resources and time. This is why it is important to relate the test effort to the expected product risks: thorough testing where great risks are discerned; light or no testing where the risks are small. The motto is: "No risk, no test"

Well-justified choices have to be made in this context. Product risk analysis (PRA) is a tool in making such choices.

The focus in PRA is on the product risks, i.e. what is the risk to the organisation if the product does not have the expected quality. This can be because the software contains defects, but also because the AO procedures are not in line with the system or the system is not sufficiently effective or too slow for the end users.

There are also (test) project risks. Examples: the system must be live on 1 January, the functional specifications are delivered too late, skilled testers are not available or the test infrastructure is not ready in time. These risks are not taken into account when establishing the test strategy, but when creating the test plan. Www.tmap.net also contains a checklist with such risks: “Test process risks”.

In practice, people are found to have highly diverging ideas of ‘risks’. A PRA requires a shared reference framework, in particular as to what product risk is understood to mean.

The Chance of failure is determined by the Chance of defects and the Frequency of use. The chance of defects is the chance that a product (component) contains a defect. The presence of a defect in the product, however, does not mean that that defect will actually manifest itself in production. When the defect is in a part of the product that is never used, the product will not fail. The more often the product component with the defect is used – or, the higher the frequency of use – the greater the chance that the product will fail. For a proper risk analysis, the separate factors of a risk are therefore taken into consideration:

• Frequency of use - In a system (component) that is accessed several dozens of times per day, the chance that a defect will manifest is many times bigger than in a system (component) that is accessed just once per year.
• Chance of defects - To assess the chance of defects, the following overview of where defects often concentrate may be of assistance. This overview is based in part on [Schaefer, 1996]:
  • Complex functions (also: complex functionality, software)
  • Completely new functions
  • Often adapted functions
  • Functions for which specific tools or techniques were deployed for the first time
  • Functions that were handed over to someone else during development
  • Functions realised under extremely high time pressure
  • Functions requiring above-average optimisation (e.g. rendering a function extremely fast or economical, often because of resource restrictions, typically either machine time or memory)
  • Functions in which many defects were found earlier (e.g. in a previous release or during earlier evaluations)
  • Functions with many interfaces.
• The chance of defects is bigger as well with:
  • Inexperienced developers
  • Inadequate commitment of users
  • Inadequate quality management during development
  • Inadequate quality of the development tests
  • New development tools and environment
  • Big development teams
  • Development teams with sub-optimal communication (due to geographical or personal causes).
• An important point of concern in relation to the chance of defects (and therefore the chance of failure) is that it can be assessed more correctly as the development or maintenance process advances. The PRA can (and will) therefore be adjusted in the course of the test process.
• Damage - What is the damage that will be suffered if the defect manifests itself? A distinction is to be made between direct and indirect damage for the organisation. Direct damage would involve loss of revenue, damage to third parties, economic loss, environmental or physical damage (especially with embedded systems - explosion, crash, car injury) costs for correction and repairs. Indirect damage would be e.g. damage to the image, loss of trust, damage claims from third parties, overload of the help desk or, for government institutions, social damage such as for the tax department (errors in tax refunds) and justice (formality issues due to which serious offenders are not condemned). Moreover, damage can result for third parties because the system does not function as intended. The damage usually grows when the defect has an impact on other functions or systems. An organisation can determine damage at various levels, such as at the level of business processes, business objectives, business requirements, but also systems or subsystems.